System and method for managing bandwidth usage rates in a packet-switched network

ABSTRACT

A computer-implemented system is disclosed for managing bandwidth usage rates in a packet switched network. The system includes one or more servers configured to execute computer program steps. The computer program steps comprises monitoring bandwidth usage rate at a first provider interface, determining if bandwidth usage rate at the provider interface exceeds a bandwidth usage rate limit; and rerouting Internet traffic from the provider interface having bandwidth that exceeds the bandwidth usage rate limit to a second provider interface having available bandwidth capacity.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 14/335,234, filed Jul. 18, 2014 which claims priority to U.S.Provisional Application No. 61/858,160, filed Jul. 25, 2013, which areall incorporated by reference herein.

FIELD OF THE INVENTION

The present invention relates to managing bandwidth usage rates, andmore particularly to a system and method for managing bandwidth usagerates in a packet-switched network.

BACKGROUND OF THE INVENTION

Multi-homed networks are often connected to the Internet through severalInternet Service Providers (ISPs). Multi-homed networks are advantageousif one of the connections to an ISP fails. Each ISP assigns an IPaddress (or range of them) to a company. As soon as a router assigned toconnect to that ISP determines that the connection is lost, it willreroute all data through one of the other routers. In order to decreaseoperational costs and improve on network reliability and performance,however, multi-homed networks require bandwidth management and control.Bandwidth management and control typically involves managing bandwidthusage rates (also known as bandwidth rates) by considering bandwidthusage rate limitations (limits). These limitations include capacitylimits, physical limits or contracted bandwidth limits (e.g., flat rateor 95th percentile).

In such multi-homed networks, bandwidth usage rates must be managed foreach provider interface within a group of provider interfaces (two ormore provider interfaces that form a group for bandwidth balancing). Aprovider interface is a hardware part of a router or network devicedesignated for connection with an Internet Service Provider's router orother hardware as known to those skilled in the art. Bandwidth usagerates may require balancing across two or more provider interfaceswithin a provider interface group for improving network efficiency,reliability and costs. The typical answer or solution for managingbandwidth rates is to manually reroute some network prefixes carrying aspecific amount of bandwidth to other provider interfaces. However, thistask is too complex and time consuming when it must be performed on anongoing or periodic basis.

It would thus be advantageous to provide a system and method that willovercome the problems described above.

SUMMARY OF THE INVENTION

A system and method is disclosed for managing bandwidth usage rates in apacket-switched network.

In accordance with an embodiment of the present disclosure, acomputer-implemented system is disclosed for managing bandwidth usagerates in a network. The system includes one or more servers configuredto execute computer program steps. The computer program steps comprisemonitoring bandwidth usage rate at a first provider interface,determining if bandwidth usage rate at the provider interface exceeds abandwidth usage rate limit, and rerouting Internet traffic from theprovider interface having bandwidth that exceeds the bandwidth usagerate limit to a second provider interface having available bandwidthcapacity.

In accordance with yet another embodiment of the present disclosure, acomputer-implemented system is disclosed for managing bandwidth usagerates in a network. The system including one or more servers havingmemory, one or more processors and one or more computer program modulesstored in the memory and configured to be executed by the one or moreprocessors, the computer program modules comprising instructions formonitoring bandwidth usage rate at a provider interface, determiningbandwidth overusages on the provider interface, whereby bandwidthoverusage is determined when a bandwidth rate on the provider interfaceexceeds a bandwidth rate limit and a protection gap, determining anumber of bandwidth overusages that exceed a number of allowablebandwidth overusages at the provider interface during an interval oftime, and adjusting a protection gap for the provider interface based onthe number of exceeding overusages to reduce the number of bandwidthoverusages to a value less than the number of allowable bandwidthoverusages.

In accordance with yet another embodiment of the present disclosure, acomputer-implemented system is disclosed for managing bandwidth usagerates in a network. The system includes one or more servers configuredto execute computer program steps. The computer program steps comprisecollecting bandwidth usage data for a plurality of provider interfaces,retrieving bandwidth usage data from the network relating to theplurality of provider interfaces, requesting for probing networkprefixes for the plurality of provider interfaces to determine networkprefixes that can be rerouted, determining if bandwidth rate at theplurality of provider interfaces exceed bandwidth rate limits,retrieving network prefix evaluation results to determine networkprefixes that can be rerouted, and applying new routes on network inaccordance with network prefixes that can be rerouted.

In accordance with yet another embodiment of the present disclosure, acomputer-implemented system is disclosed for managing bandwidth usagerates in a network. The system includes one or more servers configuredto execute computer program steps. The computer program steps comprisesforming a group of provider interfaces that may be controlled as anaggregated group of provider interfaces, calculating aggregated grouptotal limit and aggregated group usage values, and determining ifaggregated group usage values exceed group limits.

In accordance with another embodiment of the present disclosure, acomputer-implemented method is disclosed for managing bandwidth rate ina packet switched network, wherein the method is implemented in one ormore servers programmed to execute the method, the method comprisingmonitoring, by the one or more servers, bandwidth rate for a providerinterface, comparing, by the one or more servers, the monitoredbandwidth rate to a bandwidth rate limit, and determining if thebandwidth rate exceeds the bandwidth rate limit. In the embodiment, themethod further comprises monitoring, by the one or more servers, anamount of data carried by a plurality of network prefixes, selecting, bythe one or more servers, a plurality of network prefixes that carry anamount of bandwidth to be rerouted from a provider interface thatexceeds the bandwidth rate limit, determining, by the one or moreservers, a destination provider interface for rerouted bandwidth, andinjecting, by the one or more servers, a network prefix into a router.

In accordance with yet another embodiment of the present disclosure, asystem is disclosed for managing bandwidth usage rate for one or moreprovider interfaces in a network. The system includes one or moreinterconnected routers and one or more servers communicating with theone or more routers configured to execute one or more of the computerprogram modules stored in the one or more servers. The computer programmodules comprise a first module (traffic collector) for analyzingnetwork prefixes of a plurality of provider interfaces carrying anamount of bandwidth, a second module (network explorer) for determiningnetwork prefixes of the plurality of provider interfaces capable ofreceiving rerouted bandwidth, a third module (route injector) forcommunicating with the one or more routers for injecting routing changesbased on the network prefixes capable of receiving rerouted bandwidth, afourth module (stats collector) for periodic retrieval of the currentprovider interface bandwidth usage rates, and a fifth module (bandwidthcontroller) for determining if bandwidth usage rates exceed a bandwidthlimit.

In yet another embodiment of the present disclosure, a method isdisclosed for managing a controlled network including multipleconnections to an Internet Service Provider or a plurality of InternetService Providers. The method is implemented in one or more serversprogrammed to execute the method. The method comprises monitoringbandwidth usage rates at a plurality of provider interfaces, evaluatingnetwork prefixes carrying a specific amount of Internet traffic,comparing estimated bandwidth rate with the bandwidth rate limitsspecified in a central repository and dynamic bandwidth rate limitsevaluated in order to reroute network prefixes carrying a specificamount of bandwidth to prevent violation of bandwidth rate limits and toprevent network performance degradation by the provider interfacecongestion.

In accordance with this embodiment of the disclosure, a method isdisclosed for managing a controlled network, wherein the method isimplemented in one or more servers programmed to execute the method. Themethod comprises detecting whether a provider interface exceeds abandwidth rate limit, detecting network prefixes carrying a specificamount of bandwidth transmitted through the provider interface,selecting the statistics for each network prefix in the context of allproviders interfaces, omitting provider interfaces with no statistics,omitting providers interfaces with worse performance metrics, omittingprovider interfaces where current bandwidth usage rate exceeds thebandwidth rate limits, storing remaining provider interfaces in a sortedlist, selecting the first provider interface from the sorted list, andstoring the selection into the central repository.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a diagram of an example system for bandwidth ratemanagement in packet-switched network.

FIG. 2 illustrates the system shown in FIG. 1 wherein a networkcontroller is shown in greater detail.

FIG. 3 illustrates an example bandwidth usage rate, current bandwidthusage rate, bandwidth rate limit, bandwidth rate over-usage and aprotection gap of a provider interface.

FIG. 4 illustrates two graphs of an example algorithm evaluation ofInterface bandwidth usage versus time for exemplary provider interfaceswithin a provider interfaces group, wherein bandwidth usage rate limitsand protection gaps are calculated by the network controller of thesystem shown in FIG. 1.

FIG. 5A illustrates a high-level flowchart of example process steps ofthe bandwidth controller of the system in FIG. 1.

FIG. 5B illustrates a detailed flowchart of example process steps of thebandwidth controller of the system in FIG. 1.

FIG. 6 illustrates an exemplary figures in which Commit Priorities areshown for all provider interfaces that exceed bandwidth rate limits.

FIG. 7 illustrates an example implementation of the decision made by thepresent bandwidth controller in FIG. 2 in accordance with an embodimentof the present disclosure.

FIG. 8A illustrates a high-level flowchart of another example processsteps of the bandwidth controller of the system in FIG. 1.

FIG. 8B illustrates a detailed flowchart of another example processsteps of the bandwidth controller of the system in FIG. 1.

FIG. 9 illustrates a graph of percentage adjusted protection gap ofbandwidth usage rate limit per measurement and overusage count.

FIG. 10 illustrates a flowchart of example process steps of thecalculation of the adjustable protection gap.

FIG. 11 illustrates a flowchart of example process steps for controllingbandwidth for an aggregated group of provider interfaces.

FIG. 12 illustrates a graph depicting an example of (two) providerinterface bandwidth rate usage limits and aggregated group limits.

FIG. 13 is a diagram that illustrates example system process steps forcontrolling bandwidth on controlled network in FIG. 1.

FIG. 14 illustrates a graph depicting categories of trafficdistinguished by the bandwidth controller of the system in FIG. 1 over a5 minute-interval under normal circumstances.

FIG. 15 illustrates a graph depicting categories of trafficdistinguished by the bandwidth controller of the system in FIG. 1 over afive-minute interval for the fast react algorithm.

FIG. 16 illustrates a block diagram of a general purpose computer tosupport the embodiments of the systems and methods including thecomputer program modules disclosed in this application.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description, reference will be made to theaccompanying drawing(s), in which identical functional elements aredesignated with numerals. The aforementioned accompanying drawings showby way of illustration and not by way of limitation, specificembodiments and implementations consistent with principles of thepresent invention. These implementations are described in sufficientdetail to enable those skilled in the art to practice the invention andit is to be understood that other implementations may be utilized andthat structural changes and/or substitutions of various elements may bemade without departing from the scope and spirit of present invention.The following detailed description is, therefore, not to be construed ina limited sense. In addition, certain terms used herein are defined inthis disclosure. Terms that are capitalized shall have the same meaningas those terms without capitalization (in this disclosure or thedisclosure in the priority provisional application identified above).

FIG. 1 illustrates a diagram of an example system 10 for bandwidth ratemanagement in a packet-switched network in accordance with an embodimentof this disclosure. In particular, system 10 includes network controller100, two or more Internet Service Providers 103, one or more routers104, provider interfaces 106, network device 107, controlled network 105and destination network 101.

Controlled network 105 is connected to Internet Service Providers 103via routers 104. Network controller 100 is connected to (communicateswith) controlled network 105 and network device 107. Network device 107is connected to routers 104. Routers 104 are also connected to InternetService Providers 103 via provider interfaces 106. Internet ServiceProviders 103 are connected to (communicate with) destination network101 via communication paths 102 (through the Internet).

Network controller 100 includes one or more servers or other computersthat comprise a set of software modules for implementing system 10 inaccordance with embodiments of this disclosure. Network controller 100and these modules are discussed in more detail below.

Destination network 101 is the destination network in which Internettraffic is intended for delivery.

Internet Service Providers 103 are companies that offer Internet service(access) to its customers as known to those skilled in the art.

Routers 104 are components in the controlled network 105 that are usedto route data through a network as known to those skilled in the art.Routers 104 provide dynamic routing protocol support and IP trafficstatistics reporting (export). Cisco Systems, Inc., for example,manufactures and markets many routers that may be used.

Controlled network 105 comprises (1) the computer servers, routers andother components including routers 104 and network device 107 and (2)computer program modules and other software that make a network of abusiness or enterprise.

Provider interfaces 106 are a hardware part of routers 104 oralternatively network device 107. Provider interfaces 106 are used toprovide connectivity to the routers and/or other hardware components ofInternet Service Providers 103.

Network device 107 is a component within controlled network 105. Networkdevice 107 includes one or more packet switches, routers or othernetwork devices as known to those skilled in the art with the ability toduplicate traffic data. Network device 107 is used to copy all transitIP packets to traffic collector 108 as described below. (Routers 104 canfunction similarly to network device 107 and network device 107 canfunction similarly to routers 104).

Reference is now made to FIG. 2. FIG. 2 illustrates system 10 shown inFIG. 1 wherein network controller 100 is shown in greater detail. Asindicated above, there are two alternate transmission paths 102connecting controlled network 105 to destination network 101 viaInternet Service Providers 103. In order for system 10 to operate asdisclosed herein, controlled network 105 must incorporate two or morealternative transmission paths 102 to the destination network 101 toenable the proper rerouting of Internet traffic (data). In accordancewith an embodiment of the present disclosure, system 10 will not beactivated until a current bandwidth usage rate (information volume/time,e.g., Megabits per second) of the Internet Service Providers 103 exceedsthe bandwidth rate limits (e.g., bandwidth rate limit, 95th bandwidthrate limit, or limits inside a provider interfaces group) set by theoperator. That is, once the bandwidth rate at provider interface 106 isexceeded, system 10 is triggered and data is rerouted in accordance withan embodiment of the present disclosure as described in more detailbelow.

Reference is now made to network controller 100 in FIG. 2 in detail. Asindicated above, network controller 100 is a system that includes one ormore servers incorporating a plurality of computer program modules.These servers include, among other components, one or more processorsfor executing the plurality of computer program modules. An example of aserver is shown in FIG. 16. These modules include traffic collector 108,network explorer 109, stats collector 110, route injector 111, bandwidthcontroller 112, frontend 114, central repository 113. The modulesdescribed above or portions thereof may be incorporated on serversand/or other components that are part of or entirely separate fromcontrolled network 105 as known to those skilled in the art. Networkcontroller 100 is connected to controlled network 105.

Traffic collector 108 is a module that analyzes network prefixescarrying a specific amount of bandwidth by using different networkmonitoring protocols for gathering IP traffic statistics.

Network explorer 109 is a module used for determining network prefixreachability and measuring and collecting each network prefixperformance metrics such as packet loss, latency, jitter, and storesthese metrics into the central repository 113. This is referred to asprobing to distinguish from processes that determine reachability,measure and collect data for other purposes.

Stats collector 110 is a module used for monitoring and periodicretrieval of current Interface bandwidth usage rate of providerinterfaces 106.

Route injector 111 is a module used for communicating with routers 104for injecting routing changes using Border Gateway Protocol (BGP) or byother dynamic routing protocols as known to those skilled in the art.

Bandwidth controller 112 is a module used for performing analysis ofdata collected by traffic collector 108, network explorer 109 and statscollector 110 and making traffic routing decisions.

Frontend 114 is a module for interacting with operators to enable themto configure, monitor and report to an operator. That is, frontend 114is a visual interaction interface between operator and networkcontroller 100. Frontend 114 is described in more detail below. (Theoperator may be a person or automatic management, monitoring orreporting system as known to those skilled in the art.)

Central repository 113 is a module for storing module(s) configurationinformation and transferring or storing data exchanged between themodules. Central repository 113 may be a database or other storagemedium. As indicated above, controlled network 105 includes networkdevice 107. Network device 107 includes one or more packet switches,routers or other network devices with the ability to duplicate trafficdata. That is, network device 107 is used to copy all or some transit IPpackets to traffic collector 108. As indicated above, system 10 furtherincludes a plurality of Internet Service Providers 103 and correspondingprovider interfaces 106 for connecting and communicating with InternetService Providers 103. The duplicated traffic is generated by networkdevice 107 and collected by the traffic collector 108 (port mirroring).The IP traffic statistics are generated by routers 104 and collected bythe traffic collector 108. Traffic collector 108 arranges and aggregatesreceived data, and stores it into central repository 113. Examples of IPtraffic statistics appear at the rear of this description.

As indicated above, stats collector 110 periodically retrieves providerinterfaces 106 bandwidth statistics from routers 104 and network device107 and stores this data in central repository 113. The network explorer109 measures each network prefix performance metrics, such as packetloss, latency, jitter, and stores these metrics into the centralrepository 113. Bandwidth controller 112 retrieves the list of providerinterfaces 106 exceeding bandwidth rate limits from central repository113, bandwidth controller 112 also retrieves the list of networkprefixes from central repository 113 that can be rerouted from aprovider interface 106 that exceeds bandwidth rate limits to adestination provider interface 106 that has been selected for datarerouting in accordance with the present an embodiment of the presentdisclosure. Route injector 111 applies the selection made by bandwidthcontroller 112 to a single or multiple routers 104 (for Internet trafficegress).

System 10 collects provider interface 106 bandwidth statistics andstores the collected information in the central repository 113. Thebandwidth statistics include the amount of data sent and received viathe provider interfaces over a specific period of time. The value ofthis period of time is preferably 60 seconds, but those skilled in theart know that various time periods may be used for this purpose.Examples of bandwidth statistics appear at the rear of this description.

Traffic collector 108 collects IP traffic statistics, based upon networkmonitoring protocols known to those skilled in the art. An example ofsuch protocols is IP Flow Information Export (IPFIX). IPFIX is describedin “Specification of the IP Flow Information Export (IPFIX) Protocol forthe Exchange of IP Traffic Flow Information,” B. Claise, IETF RFC 5101,January 2008. Another exemplary protocol is derived from sFlow inaccordance with the protocol specifications promulgated by the sFlow.orgconsortium, netflow, or analyzing raw, duplicated traffic data. Thesestandards/specifications and protocols may be found at the InternetEngineering Taskforce (IETF) Website at www.ietf.org.

The statistics (i.e., the total amount of bytes sent in IP packets toparticular destination addresses) collected are aggregated into networkprefixes carrying a specific amount of bandwidth based on the list ofnetwork prefixes retrieved from the central repository 113 per eachprovider interface 106 separately. Traffic collector 108 calculates theCorrection Coefficient for each provider interface 106.

Correction Coefficient is the ratio of (1) provider interface currentbandwidth usage rate retrieved by stats collector 110 to (2) thestatistics collected by traffic collector 108 (i.e., CorrectionCoefficient=total bandwidth usage rate for all provider interfaces (inbytes) per X time period/total IP traffic statistics (in bytes) per Xtime period). Bandwidth data usage averages for a specific period oftime for each of the analyzed network prefixes are multiplied by theCorrection Coefficient, in order to correct the original data volumepotentially distorted by router 104 restrictions or the networkmonitoring protocol sampling rate. That is, the correction coefficientis required to restore traffic volume information on a per-networkprefix basis due to certain router's 104 restrictions or due to thesampling rate (selective traffic information collection in networkdevices.) The statistics are stored in the central repository 113 everyspecific period of time.

Traffic collector 108 updates bandwidth data usage averages for aspecific period of time for each network prefix, per provider interface,and stores the statistics in the central repository 113 for subsequentuse by bandwidth controller 112.

As indicated above, frontend 114 is an interaction interface betweenoperator and network controller 100. Frontend 114 is used forconfiguration, reporting and management purposes. Frontend 114 includesa GUI (Graphical User Interface), CLI (Command Line Interface),Statistical Reports an API (Application Programming Interface) and/orother interfaces known to those skilled in the art. As indicated above,operator can be human beings, automated systems, monitoring systems orother systems known to those skilled in the art. Operator can manage orconfigure the network controller 100 using the frontend 114, whichenables adding, editing, deleting data (or other actions) used andexchanged between the modules and the configuration parameter values.This resulting configuration information is then stored in the centralrepository 113 of network controller 100 in accordance with anembodiment of the present disclosure.

A network prefix is a part of the IP (Internet Protocol) address spacein accordance with either IP version 4 or IP version 6 as known to thoseskilled in the art. Specifically, network prefix is a network part of anIP address and network size. Data packets contain a destinationaddresses. These destination addresses are aggregated (transformed) intonetwork prefixes. Addresses can be aggregated into a fixed size (IPv4 orIPv6). Subnetting is performed against the target IP addresses tocompose the network prefix. The corresponding description for subnetsand prefixes is defined in V. Fuller, T. Li, “Classless Inter-domainRouting (CIDR)”, IETF RFC 4632.

Network controller 100 in accordance with an embodiment of the presentdisclosure is useful when at least one provider interface 106 exceedsthe bandwidth usage rate limits and there exists one or more availablealternative provider interfaces 106 selected as Destination providerinterface by the network controller 100.

FIG. 3 illustrates a bandwidth usage rate, current bandwidth usage rate,bandwidth rate limit, bandwidth rate over-usage (also referred to asoverusage) and a protection gap of a provider interface. The providerinterface current bandwidth usage rate and the bandwidth rate limits areset by the operator and stored in the central repository or areautomatically evaluated by network controller 100 in accordance with anembodiment of the present disclosure. The exceeding bandwidth rate(traffic) is subject to be rerouted to other provider interfaces. Themethod (algorithm) of network controller 100 in accordance with anembodiment of this disclosure can reroute more bandwidth rate (traffic)than the exceeded provider interface in order to keep a protection gapas seen in FIG. 3.

The protection gap is an additional bandwidth rate limit that is used toprovide a buffer for the bandwidth rate growth up to the bandwidth ratelimit without additional intervention by network controller 100, asprovider interfaces exceeding bandwidth rate limits tend to furtherincrease their bandwidth usage rate. Typically, the protection gap forprovider interface bandwidth rate limits constitutes a subtraction fromthe provider interface bandwidth rate limits or addition of 5% to thebandwidth limit for a load balanced group of provider interfaces. Theselimits are configurable by an operator or adjustable protection gaps asknown to those skilled in the art. Adjustable protection gaps arediscussed below.

The bandwidth rate over-usage is calculated by network controller 100(bandwidth controller 112) comparing provider interface currentbandwidth rate with the limits stored in the central repository 113 orcan be automatically calculated by network controller 100 (bandwidthcontroller 112) in accordance with an embodiment of the presentdisclosure, using 95th percentile calculation, algorithms fordistributing bandwidth rate limit in a provider interfaces load balancedgroup or other calculations known to those skilled in the art.

The 95th percentile calculation is done by using well-known algorithms,such as retrieving all bandwidth usage statistics, sorting ascending,ignoring the top 5% of the values. Network controller 100(modules/algorithms) in accordance with an embodiment of the presentdisclosure is not limited to a specific amount of percentiles, as othervalues can be also used.

FIG. 4 illustrates two graphs of algorithm evaluation (method) ofInterface bandwidth usage versus time for exemplary provider interfaceswithin group of load balanced provider interfaces, wherein bandwidthusage rate limits and protection gaps are calculated by networkcontroller 100 of system shown in FIG. 1. In short, FIG. 4 illustratesthe algorithm evaluation (method) of proportional bandwidth ratedistribution in a load balanced group of provider interfaces. In orderto evaluate the Proportion of bandwidth usage rate for load balancedgroup of provider interfaces, the algorithm sums the current bandwidthusage rate for each provider interface in the load balanced group ofprovider interfaces, and divides the result by the sum of each providerinterface bandwidth rate limits. In order to evaluate the bandwidth ratelimit for a particular provider interface in the load balanced group ofprovider interfaces, the evaluated Proportion is multiplied by theprovider interface bandwidth rate limit.

The formulas in FIG. 4 are used in the following example. Assumeprovider interfaces current bandwidth rate are obtained and the limitsare retrieved from central repository 113 (or calculated as 95thpercentile). In this example, two Internet Service Providers (ISP1,ISP2) are part of a load balanced group of provider interfaces. ISP1 hascurrent usage of 250 Mbps and ISP2 has 150 Mbps. ISP1 has a limit of 400Mbps and ISP has a limit of 600 Mbps. The current bandwidth ofISP1+current bandwidth of ISP2=400 Mbps. The limit of ISP1 and ISP 2equals 1000 Mbps. 400/1000=0.4 or 40%. So, with the network controller100 in accordance with an embodiment of the present disclosure, there is40% of bandwidth usage estimated on each of the provider interfaceswithin the load balanced group. ISP1 should have 160 Mbps but has 250Mbps, so 90 Mbps are exceeding the bandwidth rate and are subject torerouting to any other provider interface. ISP2 can have 240 Mbps butcurrent usage is 150 Mbps. There is thus 90 Mbps of bandwidth rateunder-usage. The network controller 100 takes into account bandwidthrate for underused provider interface+volume from the bandwidthcorrection table in order to estimate bandwidth usage and to preventpossible over-usage on the destination provider interface. This is justone example of the use of the formulas in FIG. 4 and the networkcontroller to manage and control bandwidth usage rates on providerinterfaces.

FIG. 5A illustrates a high-level flowchart of an example process stepsof the bandwidth controller of the system in FIG. 1. Execution beginswith step 400 of the method wherein the bandwidth usage rates atprovider interfaces are monitored. At step 410, the bandwidth rates atthe provider interfaces are compared with the bandwidth rates limitsstored in central repository 113. Bandwidth controller 112 thendetermines if the bandwidth usage rate at the provider interfaces 106exceeds the bandwidth usage rate limits at step 420. If so, bandwidthcontroller 112 then directs re-routing of traffic from the exceededprovider interface 106 to the alternative provider interface withavailable bandwidth capacity at step 430. Those skilled in the art knowthat additional steps may be employed or less in accordance with anembodiment of the present disclosure.

FIG. 5B illustrates a detailed flowchart of an example process steps(algorithm) of the bandwidth controller of the system in FIG. 1.Bandwidth controller 112 retrieves the list of the provider interfaces106 from central repository 113 at execution step 501. For each providerinterface, at decision step 502, the bandwidth controller 112(algorithm) checks if the current bandwidth rate limit is greater thanbandwidth rate limit set by the operator and stored in the centralrepository 113 or 95th bandwidth rate limit, or greater than bandwidthrate limit inside a provider interface group. Once the bandwidthover-usage is confirmed, the bandwidth controller 112 (algorithm)retrieves from central repository 113 the list of network prefixescarrying a specific amount of bandwidth (traffic) through thisparticular provider interface at step 503, and data is collected andaggregated by the traffic collector 108. Bandwidth controller 112(algorithm) processes each retrieved network prefix at step 504.

Execution moves to decision step 505, wherein bandwidth controller 112(algorithm) stops analyzing the list of network prefixes from theoverloaded provider interface until bandwidth usage rate is broughtbelow the protection gap. Step 505 is required to reroute only exceedingbandwidth (instead of all bandwidth).

Execution moves to step 506 wherein bandwidth controller 112 (algorithm)(1) retrieves from the central repository 113 a set of data for eachprovider interface, comprising, but not limited to, bandwidth statisticsfor the latest period analyzed by the traffic collector 108, bandwidthdata averages for a specific period of time for each of the analyzednetwork prefixes, the latest performance metrics data for each of theanalyzed network prefixes, and (2) designates (stores) commit prioritiesinto a list of candidate provider interfaces at step 507. “Candidateprovider interfaces” is a list of possible candidate provider interfacesto which the traffic may potentially be routed. A target providerinterface is chosen from this list for rerouting traffic.

Bandwidth controller 112 (algorithm) can use performance metrics such asbut not limited to packet loss, latency, jitter of the particularnetwork prefix. In order for bandwidth controller 112 (algorithm) totake performance metrics into the consideration, these metrics have tobe collected by a set of methods and algorithms interacting with thenetwork (network explorer 109). Performance metrics can be used toprevent metric deterioration by comparing the metrics for the currentroute to the metrics for each of other routers in order to block routeswith deteriorating (worse) metrics.

Commit priority is a metric set by the operator, stored in the centralrepository 113 and used in the bandwidth controller 112. In the casewhere all provider interfaces bandwidth usage rates have exceeded aconfigured limit, this metric is used to control rerouting so thatexceeded bandwidth will be routed to “commit priority” providerinterfaces with smaller metric values (up to physical or contractuallimits) rather than to those provider interfaces that have exceededbandwidth and have higher metric values.

Execution moves to step 508, wherein bandwidth controller 112(algorithm) evaluates the list of candidate provider interfaces.Bandwidth controller 112 will (1) remove the candidate providerinterfaces at step 509 if the network prefix is not accessible throughthe particular candidate provider interfaces at decision step 510, (2)remove the candidate provider interfaces at decision step 509 in casethe performance metrics are worse at decision step 511, (3) remove thecandidate provider interfaces 509 where current bandwidth usage rateexceeds the bandwidth rate limits at decision step 512. This is done toprevent further over-usage or prevent network performance degradation bythe provider interface congestion.

Execution moves to decision step 513 wherein if any of the providerinterfaces 513 are not exceeding the bandwidth rate limits, thebandwidth controller 112 (algorithm) sorts the list of the candidateprovider interfaces by performance metrics and by the proportion of thecurrent bandwidth usage rate at step 514. However, if all providerinterfaces at decision step 513 are exceeding the bandwidth rate limits,the bandwidth controller 112 (algorithm) sorts the list of the candidateprovider interfaces by commit priority and by the proportion of thecurrent bandwidth usage rate at step 515.

Execution moves to step 516, wherein bandwidth controller 112(algorithm) retrieves the first candidate provider interface from thesorted list established in step 507 as a destination provider interface,forming an improvement and stores it in the central repository at step517 for future injection, i.e., further implementation of the decision.The improvement is a set of data stored in the central repository 113,comprising, without limitation, the network prefix, provider interfaceexceeding bandwidth rate, destination provider interface, and relatedperformance metrics.

The bandwidth correction table is defined and stored in the centralrepository 113 to represent bandwidth usage rate changes for providerinterfaces based on network prefixes carrying a specific amount ofbandwidth, rerouted from the original provider interface 106 to thedestination provider interface. The bandwidth correction table storesthese results in order to estimate bandwidth usage rate for providerinterfaces taking into account all re-reroutes made between bandwidthmeasurements. An example of this appears at the rear of thisapplication.

Bandwidth controller 112 (algorithm) modifies bandwidth correction tablestored in the central repository 113 by subtracting the bandwidth dataaverages of the analyzed network prefix from the current bandwidth usagerate of provider interfaces 106 exceeding bandwidth rate limits andadding bandwidth data averages for analyzed network prefix to thecurrent bandwidth usage of destination provider interface stats values.

The bandwidth correction table is reset each time the stats collector110 retrieves provider interfaces 106 bandwidth statistics from therouters 104 and stores them into the central repository 113.

As indicated above, FIG. 5B illustrates the detailed process steps ofbandwidth controller 112. Those skilled in the art know that the listedsteps may be formulated in different order, additional steps may beincluded, or one or more steps may be removed to the same desiredoutcome.

FIG. 6 illustrates the Commit Priorities usage to reroute bandwidthover-usage from the provider interface with higher commit priority, tothe provider interfaces with lower Commit Priorities, until the providerinterface congestion limits are met. As shown, bandwidth overusage part1filled up provider interface with commit priority 1 up to its congestionlimit. Therefore, bandwidth overusage partN has been rerouted toprovider interface with commit priority 5.

FIG. 7 illustrates the implementation of the decision made by thepresent bandwidth controller 112 in FIG. 2 in accordance with anembodiment of the present disclosure. Bandwidth controller 112 storesinformation in the central repository 113 to be used by frontend 114 andfor Improvement injection by the route injector 111 to router 104 byusing (but not limited to) Border Gateway Protocol 4 (BGP-4) as definedin the Y. Rekhter, T. Li, S. Hares, “A Border Gateway Protocol 4(BGP-4)”, IETF RFC 4271; Open the Shortest Path First (OSPF) as definedin the J. Moy, “OSPF Version 2”; IETF RFC 2328; D. Savage, D. Slice, J.Ng, S. Moore, R. White, “Enhanced Interior Gateway Routing Protocol”,IETF draft-savage-eigrp-00; Command Line interface via secure shell ortelnet protocol; or Serial Port or any other protocol, port or methodfor configuring router 104, by executing router-specific configurationcommands. Further, the Improvement is propagated between routers 104 bythe dynamic routing protocols 120 (link) established between routers 104used in order to apply the implementation to the controlled network 105.

The network controller 100 in accordance with an embodiment of thepresent disclosure executes steps of the methods in which the originalnetwork prefix is split in two or more network prefixes, in order tointeract with router 104 by the dynamic routing protocol (such as butnot limited to the BGP-4, EIGRP, OSPF) and to preserve attributes andmetrics for the injected network prefix, from the original networkprefix.

FIG. 8A illustrates a high-level flowchart of another example of theprocess steps of bandwidth controller 112 of the system in FIG. 1.Execution begins with step 800 of the method wherein the bandwidth usagerates at provider interfaces are monitored. At step 802, the bandwidthrates at provider interfaces are compared with the protection gap ofbandwidth rates limits stored in central repository 113. This protectiongap is adjustable. The bandwidth controller 112 uses the adjustableprotection gap so that small bandwidth rate usage increases on theprovider interfaces do not result in its bandwidth rate limit beingexceeded. The bandwidth controller 112 then determines if the bandwidthusage rate at the provider interfaces 106 exceeds the protection gap ofbandwidth usage rate limits at step 804. Next at step 806, bandwidthcontroller 112 identifies network prefixes that can be rerouted fromprovider interfaces that exceed protection gap of bandwidth usage ratelimits to provider interfaces with available bandwidth capacity.Execution then moves to step 808 wherein the Bandwidth controller 112then directs re-routing of traffic from the exceeded provider interface106 to the alternative provider interface with available bandwidthcapacity. Those skilled in the art know that additional steps may beemployed or less in accordance with other embodiments.

FIG. 8B illustrates a detailed flowchart of example process steps(algorithm) of the bandwidth controller 112 of the system in FIG. 1. Theflowchart in FIG. 8B is similar to the flowchart in FIG. 5B except thatthe flowchart reflects the incorporation of an adjustable protection gapas described below.

Execution begins at step 810 wherein bandwidth controller 112 retrievesthe list of the provider interfaces 106 from central repository 113. Foreach provider interface, at decision step 812 the bandwidth controller112 (algorithm) checks if the current bandwidth rate is greater than theadjustable protection gap calculated by bandwidth controller 112 basedon configuration parameters set by the operator and stored in thecentral repository 113. Once the bandwidth over-usage is confirmed, thebandwidth controller 112 (algorithm) retrieves from central repository113 the list of network prefixes carrying a specific amount of bandwidththrough this particular provider interface at step 814. Bandwidthcontroller 112 evaluates only those network prefixes that have theirperformance characteristics for all provider interfaces determined inadvance by the network explorer 109 and stored in central repository113. Bandwidth controller 112 (algorithm) processes each retrievednetwork prefix at step 816.

Execution moves to decision step 818, wherein bandwidth controller 112(algorithm) stops analyzing the list of network prefixes from theoverloaded provider interface until bandwidth usage rate is broughtbelow the protection gap. Step 818 is required to reroute only exceedingbandwidth (instead of all bandwidth).

Execution moves to step 820 wherein bandwidth controller 112 (algorithm)(1) retrieves from the central repository 113 a set of data for eachprovider interface, comprising, but not limited to, bandwidth statisticsfor the latest period analyzed by the traffic collector 108, bandwidthdata averages for a specific period of time for each of the analyzednetwork prefixes, the latest performance metrics data for each of theanalyzed network prefixes, and (2) designates (stores) Commit Prioritiesinto a list of candidate provider interfaces at step 822. Candidateprovider interfaces is a list of possible candidate provider interfacesto which the traffic may potentially be routed. A target Provider ischosen from this list for rerouting traffic.

Bandwidth controller 112 (algorithm) can use performance metrics such asbut not limited to packet loss, latency, jitter of the particularnetwork prefix. In order for bandwidth controller 112 (algorithm) totake performance metrics into the consideration, these metrics have tobe collected by a set of methods and algorithms interacting with thenetwork (network explorer 109). Performance metrics can be used toprevent metric deterioration by comparing the metrics for the currentroute to the metrics for each of other routers in order to block routeswith deteriorating (worse) metrics.

Commit priority is a metric set by the operator, stored in the centralrepository 113 and used in the bandwidth controller 112. In the casewhere all provider interfaces bandwidth usage rates have exceeded aconfigured limit, this metric is used to control rerouting so thatexceeded bandwidth will be routed to “commit priority” providerinterfaces with smaller metric values (up to physical or contractuallimits) rather than to those provider interfaces that have exceededbandwidth and have higher metric values.

Execution moves to step 824, wherein bandwidth controller 112(algorithm) evaluates the list of candidate provider interfaces.bandwidth controller 112 will (1) remove the candidate providerinterfaces 106 at step 826 if the network prefix is not accessiblethrough the particular candidate provider interfaces at decision step828, (2) remove the candidate provider interfaces at decision step 826in the event the performance metrics are worse than the performancemetrics on the overloaded provider interface at decision step 830, (3)remove the candidate provider interfaces 826 where current bandwidthusage rate exceeds the bandwidth rate limits at decision step 832. Thisis done to prevent further over-usage or prevent network performancedegradation by the provider interface congestion.

Execution moves to decision step 832 wherein if any of the providerinterfaces are not exceeding the bandwidth rate limits, the bandwidthcontroller 112 (algorithm) sorts the list of the candidate providerinterfaces by performance metrics and by the Proportion of the currentbandwidth usage rate at step 834. However, if all provider interfaces atdecision step 832 are exceeding the bandwidth rate limits, the bandwidthcontroller 112 (algorithm) sorts the list of the candidate providerinterfaces by commit priority and by the Proportion of the currentbandwidth usage rate at step 836.

Execution moves to step 838, wherein bandwidth controller 112(algorithm) retrieves the first candidate provider interface from thesorted list established in step 822 as a destination provider interface,forming an Improvement and stores it in the central repository at step840 for future injection, i.e., further implementation of the decision.The Improvement is a set of data stored in the central repository 113,comprising, without limitation, the network prefix, provider interfaceexceeding bandwidth rate, destination provider interface, relatedperformance metrics.

As stated above with respect to FIG. 5B, the bandwidth correction tableis defined and stored in the central repository 113 to representbandwidth usage rate changes for provider interfaces based on networkprefixes carrying a specific amount of bandwidth, rerouted from theoriginal provider interface 106 to the destination provider interface.The bandwidth correction table stores these results in order to estimatebandwidth usage rate for provider interfaces taking into account allre-reroutes made between bandwidth measurements. An example of thisappears at the rear of this application. Bandwidth controller 112(algorithm) modifies bandwidth correction table stored in the centralrepository 113 by subtracting the bandwidth data averages of theanalyzed network prefix from the current bandwidth usage rate ofprovider interfaces 106 exceeding bandwidth rate limits and addingbandwidth data averages for analyzed network prefix to the currentbandwidth usage of Destination provider interface stats values.

The bandwidth correction table is reset each time the stats collector110 retrieves provider interfaces 106 bandwidth statistics from therouters 104 and stores them into the central repository 113.

As indicated above, FIG. 8B illustrates the detailed process steps ofbandwidth controller 112. Those skilled in the art know that the listedsteps may be formulated in different order, additional steps may beincluded, or one or more steps may be removed to the same desiredoutcome.

The algorithm used to calculate the adjustable protection gap will nowbe described. As indicated above, bandwidth controller 112 uses aprotection gap so that small increases on bandwidth rate usage on aprovider interface do not exceed its bandwidth rate limit. Theprotection gap should be chosen to optimally use the capacity of theprovider interface but ensure that the bandwidth usage rate limitviolations are kept to a minimum. That is, if the protection gap is toosmall, small increases in bandwidth rate usage will cause bandwidth rateover-usages. If the protection gap is large then bandwidth rate usageincreases will not result in bandwidth rate over-usages. However, aprovider interface will not be used to its capacity if a largeprotection gap is employed.

In accordance with an embodiment of this disclosure, the bandwidthcontroller 112 automatically adjusts the protection gap based on thenumber of past or previous bandwidth over-usages (overloads) during aparticular billing period. That is, the quantity of previous bandwidthover-usages will dictate whether the protection gap will be increased ordecreased in value. Burstable-billing typically, as known to thoseskilled in the art, specifies how many of the top highest bandwidth rateover-usages will be excused and as such will not incur any penalties. Acustomer is billed by the bandwidth rate usage at a contracted centile.If this value still exceeds bandwidth rate limit then the customerincurs penalties according to the excess over-usage.

Given the configured start day of a billing period, the samplinginterval and the centile that the customer is billed, bandwidthcontroller 112 calculates and spreads evenly the number of allowedbandwidth rate over-usages for the entire billing period. For a givenmoment in time, bandwidth controller 112 determines the number ofallowed bandwidth rate over-usages and compares it to the number ofpreviously recorded bandwidth rate over-usages. Depending on thecomparison, bandwidth controller 112 will increase or decrease theprotection gap as described in the example below.

Typical burstable billing agreements use 95th centile and 5 minutesampling (measuring) intervals. These values also mean that 1 in 20measurements (or 5 in 100) are allowed to exceed a bandwidth rate limitfor a provider interface. In another instance, an 80th centile is usedthat allows 1 in every 5 measurements to exceed bandwidth rateover-usage (non-penalized). FIG. 9 is a graph that depicts this example.Specifically, FIG. 9 illustrates a graph depicting an example of thepercentage of bandwidth usage rate limit versus measurement count. Theflow of measurements is shown and individual measurements and (nonpenalized) allowed overusages.

FIG. 10 illustrates a flowchart of example process steps of thecalculation of the adjustable protection gap. In particular, executionbegins with step 1000 wherein bandwidth controller 112 calculates themeasurement count and the number of (non-penalized) allowed bandwidthoverusages at a specific moment in time using the following formula:

Allowed Overusages=Floor(Measurement Count*(100−Centile)/100)

whereby the variables are defined as follows:

-   -   “Measurement Count”=Floor (Now-Start)/Interval. “Measurement        Count” is the number of measurement count during a billing        period up to the specific moment in time.    -   “Centile” is the configure centile value. Typically it is 95%.        FIG. 9 employs 80%.    -   “Floor” is the rounding down function.    -   “Now” is the specific moment in time in minutes.    -   “Start” is the time in minutes of the beginning of the billing        period.    -   “Interval” is the length in minutes of the sampling interval        (typically 5 minutes.)

FIG. 9 presents the number of measurements on the horizontal axis andhighlights allowed over-usages. For example, if for a specific moment intime Measurement Count is 14, then only 2 overusages are permitted asdepicted by triangles at measurement 5 and 10 on the horizontal axis.

Now, execution moves to step 1002 wherein bandwidth controller 112retrieves bandwidth usage values from the central repository 113 at thestart and desired end points of a billing period, and then determines(counts) the number of bandwidth rate overusages recorded. The trafficcollector 108 and stats (statistics) collector 110 continue to generateand store these bandwidth rate overusage values in the centralrepository 113. The central repository 113 ensures that all of itsrecordings are stored during a current billing period.

Bandwidth controller counts the number of bandwidth rate overusage forcurrent billing period as follows:

-   -   For each (Measurement between (Start, Now), if (Bandwidth        Usage>Bandwidth Rate Limit), then increment Overusage Count;        where the following variables are defined:    -   “Measurement” represents individual records stored in the        central repository 113 that include bandwidth rate usage values.    -   “Start” is the beginning of the billing Interval.    -   “Now” is the specific moment in time for which bandwidth        controller 112 makes the calculation.    -   “Bandwidth Usage” is the actual bandwidth usage recording when a        measurement is taken.    -   “Bandwidth Rate Limit” is the configured bandwidth rate limit        for the specific provider interface.    -   “Overusage Count” is the number of bandwidth rate over-usage        recorded during this billing period.

FIG. 9 also highlights bandwidth rate over-usage measurements with an“'x” on the bandwidth rate usage line. For the example, when there havebeen 14 measurements, then the number of over-usages up to measurement14 is 4.

Execution then proceeds to step 1004, wherein bandwidth controller 112determines the number of previous over-usages (Overusage Count below)that exceed the number of allowed overusages (Allowed Overusages below).If the number of previous overusages (Overusage Count) is smaller thanthe allowed overusages (Allowed Overusages), then the protection gapwill have the nominal value. The formula below reflects thisrelationship:

Excess Overusage Count=max(0, Overusage Count−Allowed Overusages)

whereby “max” is a function that returns the maximal value from a givenset of values.

For example, FIG. 9 illustrates that the Measurement Count=14 while theExcess Overusage Count=(4−2), i.e., 2.

Execution then proceeds to step 1006, wherein bandwidth controller 112calculates the protection gap. The protection gap is calculated based on(1) upper and lower protection gap limits and on the number of adjustingsteps given by the maximum number of excess overusages that it considersacceptable. These are calculated by the following formula:

ProtectionGap=LIMIT_UP−(LIMIT_UP−LIMIT_LOW)*min(1, Excess OverusageCount/MAX_ALLOWED_EXCESS).

whereby the following variables are defined:

-   -   “LIMIT_UP” is the upper level in percentage when bandwidth        controller 112 starts rerouting traffic. Typically this value is        configured to be 99%. However, a user may select other values        for LIMIT_UP as desired.    -   “LIMIT_LOW” is the lower level in percent when bandwidth        controller 112 starts rerouting traffic. This is configurable        and will typically be a value around 75-80%. However, those        skilled in the art know that other values may be selected for        LIMIT_LOW.    -   “min” is a function that returns the minimal value from a given        set.    -   “Excess Overusage Count” is the value calculated above.    -   “MAX_ALLOWED_EXCESS” is the adjustable value that helps        determine the protection gap. This value enables the excess        overusage count to be transformed into percent for the bandwidth        limits and protection gap value. If MAX_ALLOWED_EXCESS value is        5, then 5 excessive past overusages will bring the protection        gap to its lower limit. If this value is set to 50 then the 5        past overusages mentioned above set the protection gap at only        10 percent of the interval between LIMIT_UP and LIMIT_LOW. The        system sets this value at the number of overloads allowed per        day but those skilled in the art know that other values can be        employed.

For example, FIG. 9 illustrates that the protection gap percentagebetween measurement 10 and 15 is two steps down from protection gap'supper limit.

Bandwidth controller 112 calculates the protection gap value aspercentage of bandwidth rate limit. Subsequently, bandwidth controller112 uses the value to calculate a value in Mbps using the formula:ProtectionGapMbps=Bandwidth Rate Limit*Protection Gap/100%. Once theprotection gap is calculated, bandwidth controller 112 uses this valueto determine when it shall start rerouting traffic from a specificprovider interface and make the decisions based on the configuredbandwidth.

The entire algorithm is made as part of decision diamond 812 in FIG. 8Band the calculated value is reused in subsequent steps of FIG. 8B.

The discussion now relates to controlling bandwidth usage on anaggregated group of provider interfaces and reference is made to FIGS.11 and 12.

Customers have the option to deploy network configurations with aseveral provider interfaces that interconnect with a single ISP.Additional provider interfaces can be advantageous in severalcircumstances. For example, the additional provider interfaces mayprovide sufficient capacity in the event that a single providerinterface cannot fulfill very large capacity requirements or provideredundancy in the event that one provider interface becomesnon-operational. In this case, the network can revert to the remainingprovider interfaces and continue operating. In accordance with anembodiment of this disclosure, the provider interfaces may be configuredto form a group or an aggregated group. The combined behavior of theaggregated group may be controlled (by controlling the characteristicsof provider interfaces). Some of these behaviors include load balancing,optimization and bandwidth usage on the aggregated group. Bandwidthusage of the aggregated group is controlled as described below.

Bandwidth controller 112 recognizes the number of provider interfacesthat are controlled as an aggregated group. These provider interfacesare configured as a group and their characteristics of the providerinterfaces are summed (e.g., provider interface 1, provider interface 2,provider interface N). Two characteristics are used for the aggregatedgroup—bandwidth limit and current bandwidth usage rate. These arecalculated as follows:

Aggregated Group Total Limit=Bandwidth Limit PI_1+Bandwidth Limit PI_2 .. . +Bandwidth Limit PI_N.

(“PI” is the provider interface).

Aggregated Group Total Usage=Current Bandwidth Usage PI_1+ . . .+Current Bandwidth Usage PI_N.

The aggregated group characteristics are used to control bandwidthusage, for example, to calculate the protection gap or to decide if abandwidth rate overusage event has occurred. The characteristics of theaggregated group are used to make a decision in diamond 812 in FIG. 8B.The detailed workflow of this operation is depicted in FIG. 11 wherein aflowchart is illustrated of example process steps of controllingbandwidth for aggregated group provider interfaces 106.

In a particular, execution begins at step 1100 wherein providerinterfaces are retrieved from configuration established by an operator(user).

Execution proceeds to step 1102 wherein provider interfaces 106 areidentified and formed as an aggregated group for subsequent bandwidthcontrol.

Execution then proceeds to step 1104 wherein the aggregated group totalbandwidth limit and aggregated usage values are calculated.

Execution then proceeds to step 1106 wherein aggregated group usagevalues are compared to group limits and it is determined if such valuesexceed such limits. Execution then ultimately proceeds to step 812 inFIG. 8B. The remaining steps are the same as in FIG. 8B. Therefore,these steps will not be repeated here.

FIG. 12 illustrates a graph depicting an example of characteristics oftwo provider interfaces whereby (1) provider interface 1 and providerinterface N limits that are used to calculate the overall AggregatedGroup Total Limit, (2) provider interface 1 and provider interface Nbandwidth usage are used to calculate aggregated group total usage, (3)aggregated group total limit and aggregated group total usage are theresulting values of above two operations, (4) aggregated groupoverusages highlight the events when actual aggregated group total usageexceeded aggregated group total limit and (5) aggregated groupprotection gap is calculated based on the aggregated group total limit.Aggregated group usage exceeds the aggregated group protection gap andaggregated total limit (100 Mbps) four times (designated by an “x”).FIG. 12 highlights that multiple detected (i.e., registered) overusageson provider interface 1 (starting at measurements 4, 5 etc.) arecompensated by underusages on provider interface N. Only when thebandwidth of overusages exceed the bandwidth of underusages bandwidthcontroller 112 will attempt to re-route some traffic towards providerinterfaces that are not part of the group and will consider asoverusages measurements that exceeded the aggregated group total limit.

The discussion now turns to an algorithm used to improve system reactiontime to prevent bandwidth overusages (fast react algorithm).

As known to those skilled in the art, meters that take bandwidth usagemeasurements on provider interfaces (in controlled network 105) and anISP are sampling every 5 minutes. The Multi Router Traffic Grapher(MRTG) is one example of a metering tool (http://oss.oetiker.ch/mrtg/)widely used by those skilled in the art. Bandwidth rate overusages arerecorded as an actual overusage when bandwidth usage values measured bythe meter exceed configured bandwidth rate limits on a providerinterface 106. However, if bandwidth controller 112 is capable ofrerouting exceeding traffic with sufficient speed, the bandwidth usagerate measurement will not register a bandwidth rate overusage.

It is assumed that for a previous measurement interval, acceptablebandwidth rate usage values have been observed or in the event thatbandwidth rate usage was excessive, then bandwidth controller 112already took the possible actions in order to reduce it. According tothis assumption, it is also assumed that the measurement by a meter istaken at a midway point during a measurement interval (i.e., afterapproximately 150 seconds). If the measurement by the meter is takenearlier, then the new flows increase less significantly over time. Thus,bandwidth rate usage has a smaller chance to exceed an enforcedprotection gap. If the measurement by a meter is taken after the midwaypoint in the measurement interval, then bandwidth controller 112 andcontrolled network 105 have a greater amount of time in which to reroutetraffic away from overloaded provider interfaces.

Thus, in accordance with an embodiment of this disclosure, bandwidthcontroller 112 is configured to recognize that (1) current bandwidthusage is measured by the meter in 5 minute intervals, and themeasurement will be taken at a midway point during the interval, (2)there are a number of flows that a) cannot be rerouted due toperformance deterioration, b) have been recently rerouted and it is notdesirable to repeatedly reroute them since this introduces instabilityin the routing table, c) are scheduled for probing by network explorer109 or are being probed but the results are not yet set, d) have beenprobed by network explorer 109, but the results are not sufficientlytimely (outdated) to consider, or e) carry an insignificant andirrelevant bandwidth volume at the specific moment, (3) these flows arerepresented as a “cannot reroute” category (as seen in FIGS. 14 and 15),(4) network explorer 109 has established network prefix performancecharacteristics (i.e., probed) many of the active flows on the providerinterface. These are depicted by the “probed in the past” category (asseen in FIGS. 14 and 15). These are the first candidate network prefixesthat will be considered to be rerouted from the overloaded providerinterface to candidate provider interfaces. The risk is that some of thenetwork prefixes have changed their performance characteristics sincethe probes were made in the recent past, (5) network explorer 109 willfinish a few more probes shortly. These flows are grouped under “probewill finish soon” category (as seen in FIGS. 14 and 15), and (6) the newflows, depicted by the white area in FIGS. 14 and 15 have just beenscheduled for probing by network explorer 109. Given the lengthy probinginterval these flows, while carrying a significant amount of trafficwill not be probed in sufficient time to take actions during currentmeasurements cycle. Bandwidth controller 112 can decide to increasetheir measuring (probing) priority. The results of measuring (probing)will be used in the future as described below.

The configuration discussed above is reflected in the flow diagram inFIG. 13 which illustrates example system process steps for controllingbandwidth on controlled network 105. In FIG. 13, the process steps areessentially divided into two sets of steps or system actions (“A” and“B”). The sets work together, but ultimately function independently. Thesystem actions share the operation of central repository 113. In thisway, delays in execution of steps from one set of steps will generallynot affect the proper execution of the steps of the other set. This isdescribed in more detail below.

Execution begins at step 1300 wherein controlled network 105 collectsbandwidth usage data including total bandwidth usage for each providerinterface.

Execution then moves to step 1302 wherein traffic collector 108retrieves bandwidth usage of provider interface 106, passes this datafor subsequent analysis including queuing for probing of relevantnetwork prefixes. Only network prefixes that are not already queued arequeued for probing. These probing queues are stored in centralrepository 113. These network prefixes will be evaluated by networkexplorer 109 in the queued order and the results will become availableonly in the future and does not influence current cycle reaction times.

As indicated above, set “B” steps are processed simultaneously with thesteps under set “A.” Specifically, network explorer 109 probes queuednetwork prefixes stored in central repository 113 and stores suchprobing results in central repository 113. It is important to note thatnetwork explorer 109 operates continuously throughout all bandwidthcontrol cycles (wherein a bandwidth control cycle represents theexecution steps under set “A” and “B”). That is, network explorer 109probes queued network prefixes continuously. The results are stored incentral repository 113 at step 1310. Because this step is traditionallya time consuming part of the process, the box is enlarged to representsignificant time consumption as compared to the other execution stepboxes. Traffic collector 108 feeds the probing queue during previousbandwidth control cycles.

Execution proceeds to step 1304 wherein bandwidth controller 112determines if provider interface 106 exceeds bandwidth rate limit,retrieves probing results (prefixes) from central repository 113 asshown, and determines the network prefixes that can be rerouted. Thesenetwork prefixes used for rerouting are stored in central repository113. In other words, bandwidth controller 112 is given an earlyopportunity to recognize a potential overload. The analysis is donebased on existing probe results within central repository 113 withoutpriority scheduling. However, increased probing priorities may be sentfor storage in central repository 113 as shown in FIG. 13. (As indicatedabove, network explorer 109 continuously probes queued network prefixesand stores such probing results in central repository 113.) At thisstage, bandwidth controller 112 may not possess complete data about allrelevant active flow performance characteristics, and some probe resultsmay be outdated. However, a transitory analysis cycle may retrieveadditional probe results during subsequent cycles.

Execution then proceeds to steps 1306 and 1308 wherein the routeinjector 111 announces rerouted network prefixes to controlled network105 and controlled network 105 propagates and applies new routes on itsnetwork (e.g., routers, switches and other devices that form controllednetwork 105), thus rerouting traffic from overloaded provider interfaces104 to provider interfaces 104 that have available bandwidth capacity.In short, controlled network 105 has the opportunity to enforce thererouting decisions it received from route injector 111.

In this embodiment of the disclosure, network controller 100 will reactwith greater speed to potential overloads. Execution of the steps in set“A” (bandwidth control cycle) are not affected by delays in execution ofthe steps in set “B” (probing queued network prefixes). Therefore,execution bandwidth control cycle times are reduced, therebysignificantly increasing the chance that a bandwidth overusage event hasbeen addressed when a measurement by a meter is taken.

The bandwidth usage cycles repeat continuously. New probing resultsbecome available. In the event that traffic overload conditionsre-appear, action will be taken again. By reacting more rapidly, thesystem is able to adjust to fluctuating conditions and thus bring thenetwork towards desired state in smaller and more frequent steps.

FIG. 14 illustrates a graph depicting categories of trafficdistinguished by bandwidth controller (fast react algorithm) over a 5minute-interval under normal circumstances. FIG. 15 illustrates a graphdepicting the same categories of traffic over the five-minute intervalfor the fast react algorithm. When bandwidth usage for a Providerinterface 106 exceeds the protection gap value, bandwidth controller 112takes action by rerouting network prefixes. Such an event is depicted asmoment 0 (zero) in time in FIGS. 14 and 15. The dotted rectangle depictsthe time in which a traffic measurement is taken by a metering tool.

As shown in FIG. 14, the use of a cycle without fast react was able toreact by minute 4 and the meter measurement registers a bandwidth rateoverusage. However, as seen in FIG. 15, the fast react bandwidth usagecycle was carried out two rounds of rerouting actions, thus bringing thebandwidth rate usage values down by the time the meter measurement istaken.

FIG. 16 illustrates a block diagram of a general purpose computer 1600to support the embodiments of the systems and methods disclosed in thisapplication. In a particular configuration, computer 1600 may be acomputer server as described above that is configured to enable part orall of the execution of the computer program modules (software) orapplication steps in the embodiments disclosed above. The computer 1600typically includes at least one processor 1602 and system memory 1604(volatile RAM or non-volatile ROM). The system memory 1604 may includecomputer readable media that is accessible to the processor 1602 and mayinclude instructions for execution by processor 1602, an operatingsystem 1606 and one or more application platforms 1608, such as Java andone or more modules/software components/applications 1610 or partsthereof. The computer will include one or more communication connectionssuch as network interfaces 1612 to enable the computer to communicationwith other computers over a network, storage 1616 such as a hard drives,video cards 1614 and other conventional components known to thoseskilled in the art. This computer 1600 typically runs Unix or Microsoftas the operating system and include TCP/IP protocol stack (tocommunicate) for communication over the Internet as known to thoseskilled in the art. A display 1620 is optionally used.

Examples Disclosed Above IP Traffic Statistics—Example 1

-   -   Sample of “Duplicated Data” and for sFlow sampled data:    -   Decoding IP packet header:    -   Source IP address: 10.0.0.1    -   Destination IP address: 192.168.0.1    -   Protocol: TCP    -   Source port: 1024    -   Destination port: 80    -   packet size: 1500

IP Traffic Statistics—Example 2

-   -   Sample of NetFlow data:    -   Source IP address: 10.0.0.1    -   Destination IP address: 192.168.0.1    -   Protocol: TCP    -   Source port: 1024    -   Destination port: 80    -   packets: 1000    -   octets: 879000

Octets are measured in bytes and the same as the packet size from thefirst example. Source and destination IP addresses are extracted inorder to determine if this data should be analyzed, network prefix arecomputed from Destination IP address and packet sizes are summed inorder to collect bandwidth statistics.

Bandwidth Statistics—Examples

-   -   Measurement time: 11:00:00    -   Bytes: 1000000100000    -   Next measurement:    -   Measurement time: 11:01:00    -   Bytes: 1000050100000    -   (Last-Previous) *8/60=bits per second−or Bandwidth statistics.    -   Where 8 is 8 bits=1 byte    -   60=seconds between measurements.

Bandwidth statistics represented by the constantly increasing valuesretrieved by the stats collector 110 from the routers 104 by usingSimple Network Management Protocol (SNMP).

Bandwidth Correction Table—Example

-   -   The Interface 1 bandwidth usage limit is 101 Mbps.    -   At 11:00 the Interface 1 bandwidth usage rate measured to 100        Mbps.    -   At 11:01 the network controller 100 reroutes 5 Mbps from        interface 1 to any other interface.    -   The bandwidth correction table now contains −5 for Int. 1 and +5        for other interface.    -   Next—desire to reroute 5.5 Mbps TO the Interface 1:    -   Last measurement−100 Mbps+correction (−5 Mbps)=95 Mbps.+5.5 Mbps        which it is desired to reroute to the Interface 1=100.5 Mbps. Is        100.5 Mbps a greater than usage limit (101 Mbps)? Now, traffic        can be re-routed and Bandwidth Correction Table now contain        −5+5.5=+0.5 Mbps for the Interface 1.

It is to be understood that the disclosure teaches examples of theillustrative embodiments and that many variations of the invention caneasily be devised by those skilled in the art after reading thisdisclosure and that the scope of the present invention is to bedetermined by the claims below.

We claim:
 1. A system for managing bandwidth usage rates in a networkthat includes a plurality of provider interfaces configured to enableone or more routers to communicate with one or more Internet serviceproviders, the system including one or more servers configured tocommunicate with the one or more routers, the one or more servers havingmemory for storing computer program instructions and one or moreprocessors coupled to the memory, the one or more processors configuredto execute the computer program instructions, the computer programinstructions comprising: collecting data statistics relating to dataflows for each provider interface of the plurality of providerinterfaces; identifying network prefixes for each provider interfacethat carry Internet traffic based on the data statistics collected; anddetermining network prefixes from the network prefixes identified thatcan be reassigned from a first provider interface to one or morecandidate provider interfaces capable of receiving rerouted Internettraffic.
 2. The system of claim 1 wherein the computer programinstructions further comprise aggregating bandwidth usage data from thedata statistics for each network prefix identified.
 3. The system ofclaim 1 wherein the computer program instructions further determining ifbandwidth usage rate at the first provider interface exceeds a bandwidthusage rate limit.
 4. The system of claim 1 wherein the computer programinstructions further comprise measuring performance metrics of thenetwork prefixes of each provider interface of the plurality of providerinterfaces.
 5. The system of claim 3 wherein the computer programinstructions further comprise: selecting a second provider interfacefrom the one or more candidate provider interfaces for receivingrerouted Internet traffic associated with the network prefixes that aredetermined as capable of being reassigned; and rerouting Internettraffic from the first provider interface having bandwidth that exceedsthe bandwidth usage rate limit to the second provider interface of theone or more candidate provider interfaces having available bandwidthcapacity.
 6. The system of claim 3 wherein the computer programinstructions further comprise determining if the bandwidth usage rate atthe first provider interface exceeds a protection gap.
 7. The system ofclaim 6 wherein the protection gap is adjusted based on bandwidthmeasurements at the first provider interface.
 8. The system of claim 4wherein the computer program instructions further comprise determiningwhether to remove an undesirable candidate provider interface from theone or more candidate provider interfaces (1) if a network prefix of thenetwork prefixes that are determined as capable of being reassigned isnot accessible through the undesirable candidate provider interface, (2)if the undesirable candidate provider interface performance metrics areworse than the first provider interface, (3) if current bandwidth usagerate for the network prefix determined as capable of being reassignedexceeds the bandwidth usage rate limit or (4) if a bandwidth usage rateat the undesirable candidate provider interface will exceed a bandwidthusage rate limit, if a network prefix is reassigned to the candidateprovider interface.
 9. A system for managing bandwidth usage rates in anetwork, the system including one or more servers having memory forstoring computer program instructions and one or more processors coupledto the memory, the one or more processors configured to execute thecomputer programs instructions, the computer program instructionscomprising: retrieving bandwidth usage data from the network relating toa plurality of provider interfaces; identifying network prefixes thatcarry Internet traffic on each of the plurality of provider interfacesbased on the bandwidth usage data; probing the network prefixesidentified to determine performance of each provider interface of theplurality of provider interfaces; determining if bandwidth rates at theplurality of provider interfaces exceed bandwidth rate limits;identifying a first provider interface of the plurality of providerinterfaces that has a bandwidth rate that exceeds a bandwidth ratelimit; retrieving network prefixes and bandwidth usage data at the firstprovider interface; identifying at least one candidate providerinterface of the plurality of provider interfaces, based on theperformance of each provider interface, having available bandwidth thatmay receive Internet traffic associated with the network prefixesretrieved; and analyzing probing results for the retrieved networkprefixes to determine the network prefixes on the first providerinterface that can be rerouted to the at least one candidate providerinterface.
 10. The system of claim 9 wherein the computer programinstructions comprises applying new routes on the network in accordancewith the retrieved network prefixes that can be rerouted.
 11. The systemof claim 10 wherein the computer program instructions further comprisingannouncing the retrieved network prefixes that can be rerouted in thenetwork.
 12. A system for managing bandwidth usage rate for one or moreprovider interfaces in a network that includes one or more routers, thesystem including one or more servers communicating with the one or morerouters, the one or more servers having memory for storing computerprogram instructions and one or more processors coupled to the memory,the one or more processors configured to execute the computer programinstructions, the computer program instructions comprising; retrieving alist of provider interfaces in the network from a repository; collectingdata statistics relating to active flows for each provider interfacelisted; identifying network prefixes from each provider interface withinthe data statistics collected; determining network prefixes from thenetwork prefixes identified that can be reassigned from a first providerinterface to one or more candidate provider interfaces capable ofreceiving rerouted Internet traffic.
 13. The system of claim 12 whereinthe computer program instructions further comprise: aggregating usagedata from the data statistics for each of the network prefixesidentified; and determining if bandwidth usage rate exceeds a bandwidthlimit on at least one provider interface on the list of providerinterfaces.
 14. The system of claim 12 wherein the computer programinstructions further comprise probing network prefixes at each providerinterface on the list of provider interfaces to determine network prefixperformance of each provider interface.
 15. The system of claim 12wherein the computer program instructions further comprise selecting asecond provider interface from the one or more candidate providerinterfaces for receiving rerouted Internet traffic associated with thenetwork prefixes that are determined as capable of being reassigned fromthe first provider interface.
 16. The system of claim 15 wherein thecomputer program instructions further comprise communicating with theone or more routers for injecting routing changes to the second providerinterface based on the network prefixes determined capable of receivingrerouted Internet traffic.
 17. The system of claim 14 wherein thecomputer program instructions further comprise determining whether toremove a candidate provider interface from the one or more candidateinterfaces for receiving rerouted Internet traffic (1) if a networkprefix of the network prefixes that are determined as capable of beingreassigned is not accessible through the undesirable candidate providerinterface, (2) if the undesirable candidate provider interfaceperformance metrics are worse than the first provider interface, or (3)if current bandwidth usage rate for the network prefix determined ascapable of being reassigned exceeds the bandwidth usage rate limit.